Skip to content

Push hugging face storage resolution into vortex-cloud rather than vortex-python - #9199

Open
robert3005 wants to merge 3 commits into
developfrom
claude/vortex-hugging-face-format-1asxf3
Open

Push hugging face storage resolution into vortex-cloud rather than vortex-python#9199
robert3005 wants to merge 3 commits into
developfrom
claude/vortex-hugging-face-format-1asxf3

Conversation

@robert3005

@robert3005 robert3005 commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

We should treat hf:// urls as any other object store, right now we are keeping all of it in vortex-python which is not quite as versatile as having a full fledged object store.

fixes #5379

robert3005 and others added 2 commits August 4, 2026 15:28
…e registry

Vortex could not read from the Hugging Face Hub. `hf://` was understood only by
`vortex.datasets`, which parsed the URI in Python and rewrote it into an
authenticated `HTTPStore`, so `vx.open("hf://...")` failed and Polars, DuckDB and
DataFusion could not open Hub URLs at all.

A Hub repository is a set of files behind an HTTP endpoint that honours range
requests, so serving one needs no cloud SDK: an `object_store::http::HttpStore`
rooted at the repository's `resolve` prefix, carrying a bearer token when one is
available, is the whole implementation. Reads therefore keep every
`ClientOptions` setting a caller passes -- timeouts, retries, proxy
configuration, `allow_http` -- unlike the OpenDAL-backed schemes, whose bridge
owns its own HTTP client.

Unlike every other scheme in the registry, the store is rooted at a
(repository, revision) pair rather than at the URL authority, since both occupy
URL path segments. It therefore reports the in-repository path itself and lets
the registry mount it that deep; getting this wrong would send the repository
name to the Hub as part of the file path, so it is covered from both the
build-and-cache and the cached-store branch.

Listing is not supported. The Hub does not implement WebDAV `PROPFIND`, which is
how `object_store`'s HTTP store lists a prefix. Opening a known path -- `head`
plus ranged `get`, both plain HTTP -- is what a Vortex scan needs, and callers
that must expand a glob can list through the Hub's own API first.

The `hf` feature adds no cloud SDK, only `http` and `percent-encoding`, so
`vortex-python` enables it unconditionally rather than making it opt-in the way
`opendal` is.

Towards #5379.

Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Lc5zw7Le2T3pakDEUdKTYd
…reads `hf://`

`vortex.datasets` built Hub URLs and authorization headers by hand because the
reader could not resolve `hf://` itself. It can now, so the URL building
(`hf_hub_url`), the saved-login lookup (`get_token`), the endpoint-relative path
rewriting, and the separate prefix-rooted store for revisions containing `/` all
collapse into emitting `hf://` URIs and letting the reader resolve them.

Two cases an `hf://` URI cannot express keep a store of their own, both rooted at
the encoded `resolve` prefix:

* a `token` string, which the reader has no way to see;
* `token=False`, which must suppress the credentials the reader would otherwise
  read from `HF_TOKEN` or the saved login. Routing this through `hf://` would
  have quietly authenticated a read the caller asked to be anonymous, so it is
  covered by its own test.

`token=True` asks for exactly the saved login the reader already finds, so it
joins the default on the `hf://` path.

Listing stays here. The Hub serves no listing over the object-store protocol, so
expanding a glob or a directory still goes through the Hub API, which is why
`_parse_hf_uri` and the `HfApi` call remain. File paths are percent-encoded on
their way into the URI, since they become URI segments; Vortex decodes them back
into the object key.

Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Lc5zw7Le2T3pakDEUdKTYd
@codspeed-hq

codspeed-hq Bot commented Aug 5, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 12.49%

❌ 2 regressed benchmarks
✅ 1893 untouched benchmarks
⏩ 43 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation decompress[u64, (10000, 4)] 311.8 µs 361.9 µs -13.86%
Simulation decompress[u64, (1000, 16)] 65.2 µs 73.3 µs -11.1%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing claude/vortex-hugging-face-format-1asxf3 (80f90fc) with develop (88653d3)

Open in CodSpeed

Footnotes

  1. 43 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@robert3005 robert3005 added the changelog/feature A new feature label Aug 5, 2026
…x.open`

`vortex.datasets` still hand-built a Hugging Face `resolve` URL and an
authorization header for the two cases an `hf://` URI cannot express: a token
held in a variable, and a read that must stay anonymous despite the environment
offering credentials. Expose the store class those cases actually want.

`HfStore(repo_id, *, repo_type, revision, token, endpoint)` wraps the `HfConfig`
/ `make_hf_store` pair the way `CosStore` wraps `CosConfig`. `token` follows
`huggingface_hub`'s own convention, so a caller can pass its value straight
through: `None`/`True` take whatever the environment offers, `False` forces an
anonymous read, and a string is used directly. `revision` is passed literally --
unlike in a URL, the store percent-encodes a revision containing `/` itself.

`vx.open` previously accepted only the `pyo3-object_store` classes, which is why
`datasets.py` carried a comment that `CosStore` could not be read from. It now
takes the same `AnyVortexStore` extraction `read_url`/`write` use, so `HfStore`
works there -- and `CosStore` does too, removing that limitation.

Two things this surfaced:

* `HfConfig` built `ClientOptions::default()`, ignoring `ALLOW_HTTP`. Every other
  scheme picks it up because `parse_url_opts` reads its configuration from the
  environment, so a plain-HTTP `HF_ENDPOINT` -- a self-hosted Hub, or a test
  double -- behaved inconsistently. It is honoured now, which is what lets the
  new end-to-end read test exercise the real path against a local server.
* `HfRepoType` gained `FromStr` so the binding can take `repo_type="dataset"`,
  accepting the plural the Hub spells in its URLs as well.

With the store class in place, `datasets.py` no longer knows the Hub endpoint at
all: the `huggingface_hub.constants` import and the last `HTTPStore` use are gone.

Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Lc5zw7Le2T3pakDEUdKTYd
@robert3005
robert3005 force-pushed the claude/vortex-hugging-face-format-1asxf3 branch from 4014ea0 to 80f90fc Compare August 5, 2026 19:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Vortex should support reading from the Hugging Face Datasets API

1 participant